A new manner of computer interaction is now in its infancy. The words "virtual environment" or "virtual reality" will soon be commonplace. A virtual environment is an environment where some portion of the environment is artificially simulated, most often via a computer. A computer may create a graphic simulation of an environment, complete with graphic images of chairs, windows, doors, walls, etc., and even images of other people. The computer may also simulate environmental sounds. The generated objects may be viewed on a common two dimensional display, such as a computer screen, or, by viewing with special stereoscopic equipment, the objects may be made to appear three dimensional.
The most natural way for an individual to interact in a virtual environment is to directly control a graphical representation of himself. For example, if the individual turns his head, the display screen at which he is looking is appropriately updated. Also, if the individual reaches out and closes his hand, the computer generated image of his hand on the screen reaches out and closes. Such virtual environments have been discussed in the literature.
To create the sensation of a virtual reality, the computer should be able to generate and manipulate graphic images of real or imaginary objects in real time. Although generating a graphic representation of an environment may be time consuming and non-trivial to implement, much of the theory has been explored and is well-understood by those skilled in the art of interactive 3D computer graphics and solid modeling. The invention described here pertains to the important related area in which relatively little research has been done, i.e., "How may a human user perceive grasping force and texture from his computer generated counterpart in the virtual environment?"
There are many peripheral devices which have been created to allow a user to enter information into the computer. The most notable of these is the standard QWERTY keyboard. Besides the numerous modifications of this "key input" concept, there are many other devices with their associated permutations. A partial list of such devices includes mice, joy-sticks, trackballs and Computer Aided Design (CAD) tablets. The main drawback of these computer input devices is that they don't permit human users to enter information in a manner which may be the most efficient and natural. For example, in a CAD software program, the human designer may wish to rotate a 3-D graphic representation of a block on a computer screen to view and modify the hidden side. Using currently available input devices, the designer must select the axis or a sequence of axes about which the object must be rotated to achieve the desired orientation and view. After the desired aids is selected, the amount of angular rotation must be determined, usually by the linear motion of a mouse or by entering the desired amount of rotation as a decimal quantity via the keyboard. This whole procedure seems very awkward and unintuitive when compared to what a person would normally do when confronted with a similar task in the "real world," i.e., he would simply reach out, pick up and rotate the object! Providing feedback for this more natural approach to object/environment interaction is an object of this invention.
Instrumented gloves which provide finger position information to the computer have been used to manipulate simulated objects in virtual environments. Such gloves have also been used in telerobotics to control highly dextrous end-effectors to grasp real objects. However, lack of force feedback to the glove wearer has reduced the effectiveness of these open-loop manipulation approaches. Imagine a 3-D graphic model of an egg on a computer screen. Suppose you are wearing a glove which maps your finger and hand motions to a graphic image of a hand on the same screen as the egg. As you move your hand and fingers, the corresponding graphic images of the hand and fingers move in a similar manner. The task is to move your own hand and fingers to control the graphic hand on the computer screen to pick up the egg. To accomplish this task you must provide enough force to reliably grasp and lift the virtual egg, but not so much force such that the egg is crushed. Without some kind of grasping force and tactile feedback, this task would be extremely difficult.
Attempts have been made to provide information about simulated contact with virtual or telemanipulated objects to senses other than the corresponding tactile senses. One method of simulated feedback which has been tested uses audible cues. For example, the computer may beep when contact is made. Another simple method is to highlight the object once contact is made. Both these methods will require the user to re-learn hand-eye coordination. It may be frustrating and time consuming for the user to learn one of these "unnatural" methods of grasping an object, and the sensation of interacting in a virtual environment will be reduced.